Reconsidering Mutual Information Based Feature Selection: A Statistical Significance View

Abstract

Mutual information (MI) based approaches are a popular feature selection paradigm. Although the stated goal of Mi-based feature selection is to identify a subset of features that share the highest mutual information with the class variable, most current Mi-based techniques are greedy methods that make use of low dimensional MI quantities. The reason for using low dimensional approximation has been mostly attributed to the difficulty associated with estimating the high dimensional MI from limited samples. In this paper, we argue a different viewpoint that, given a very large amount of data, the high dimensional MI objective is still problematic to be employed as a meaningful optimization crit..